Conversation
romanlutz
force-pushed
the
romanlutz/add-or-bench-dataset
branch
from
fea917e to
5b341d2
Compare
There was a problem hiding this comment.
Pull request overview
Adds a new remote dataset loader for the HuggingFace OR-Bench benchmark (bench-llm/OR-Bench) so it can be discovered via SeedDatasetProvider and loaded as SeedDataset seeds, with support for both the or-bench-hard-1k and or-bench-toxic configurations.
Changes:
- Introduces
_ORBenchDatasetremote loader that fetches OR-Bench from HuggingFace and converts rows intoSeedPrompts. - Registers the new loader for automatic discovery and documents the new dataset name in the datasets loading notebook output.
- Adds unit tests covering default loading and the toxic config path.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
pyrit/datasets/seed_datasets/remote/or_bench_dataset.py |
Implements the OR-Bench HuggingFace-backed dataset loader and maps records into SeedPrompts. |
pyrit/datasets/seed_datasets/remote/__init__.py |
Imports/exports _ORBenchDataset to trigger provider registration and expose it from the remote loaders package. |
tests/unit/datasets/test_or_bench_dataset.py |
Adds unit tests validating prompt mapping and config propagation to the HuggingFace fetch helper. |
doc/code/datasets/1_loading_datasets.ipynb |
Updates the displayed list of available datasets to include or_bench. |
romanlutz
force-pushed
the
romanlutz/add-or-bench-dataset
branch
from
5b341d2 to
264aec8
Compare
| " 'ml_vlsu',\n", | ||
| " 'mlcommons_ailuminate',\n", | ||
| " 'multilingual_vulnerability',\n", | ||
| " 'or_bench',\n", |
There was a problem hiding this comment.
The notebook’s SeedDatasetProvider.get_all_dataset_names() output lists or_bench, but the loaders added in this PR register dataset names or_bench_80k, or_bench_hard, and or_bench_toxic. Update the notebook output (re-run/adjust) or align the loader dataset_name values so the documented dataset name(s) match what users can actually request.
| " 'or_bench',\n", | |
| " 'or_bench_80k',\n", | |
| " 'or_bench_hard',\n", | |
| " 'or_bench_toxic',\n", |
| "_LibrAIDoNotAnswerDataset", | ||
| "_MedSafetyBenchDataset", | ||
| "_MLCommonsAILuminateDataset", | ||
| "_ORBenchDataset", |
There was a problem hiding this comment.
__all__ includes _ORBenchDataset, but this module does not import or define that symbol anywhere (and there is no _ORBenchDataset class in the codebase). This will break from pyrit.datasets.seed_datasets.remote import * and any consumer expecting _ORBenchDataset to exist. Remove _ORBenchDataset from __all__, or add/rename the intended loader class so the exported name is valid.
| "_ORBenchDataset", |
Add remote dataset loader for OR-Bench (bench-llm/OR-Bench), an over-refusal benchmark that tests whether language models wrongly refuse safe prompts. Supports both or-bench-hard-1k and or-bench-toxic configurations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…empty categories Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Each OR-Bench config gets its own loader class with a custom description, sharing common fetch logic via _ORBenchBaseDataset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…afety_tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
romanlutz
force-pushed
the
romanlutz/add-or-bench-dataset
branch
from
50762a2 to
014b274
Compare
Add remote dataset loader for OR-Bench (bench-llm/OR-Bench), an over-refusal benchmark that tests whether language models wrongly refuse safe prompts. Supports both or-bench-hard-1k and or-bench-toxic configurations.